1. Field of the Invention
The present invention relates to an image processing apparatus which uses an estimation operator to estimate a spectroscopic property, and to a reliability evaluation method and a computer program product of the image processing apparatus.
2. Description of the Related Art
As a physical quantity which shows a physical property inherent to an object, there is a spectrum of a spectral transmittance. A spectral transmittance is a physical quantity which shows a ratio of transmitted light to incident light at each wavelength and is, as being different from color information such as an RGB value depending on a change of illumination light, information whose value does not change due to an exogenous influence and which is inherent to an object. Therefore, the spectral transmittance is used in various fields as information for reproducing a color inherent to the object itself. For example, a technology of estimating the spectral transmittance as one example of spectroscopic properties is utilized for an analysis on an image capturing a specimen in a field of a pathological diagnosis using an organism tissue specimen, specifically a pathological specimen.
In the pathological diagnosis, it is widely prevalent, for obtaining various findings, to use a microscope to perform an observation through a magnification after a block specimen obtained by an organ extraction or a pathological specimen obtained by a needle biopsy is sliced into thin sections whose thickness each is approximately several micrometers. In particular, a transmission observation using an optical microscope is one of the most common observation methods for the reason, besides the fact that its equipment is relatively inexpensive and can be easily handled, that the transmission observation has been performed historically from a long time ago. In this case, since the sliced specimen hardly absorbs and disperses light and is almost transparent and colorless, the sliced specimen generally undergoes staining with a stain prior to the observation.
Though there have been various kinds of proposals for a staining method and the total number of the kinds reaches a hundred or more, a hematoxylin-eosin staining in which two of a hematoxylin whose color is bluish violet and an eosin whose color is red are used as a stain (hereinafter referred to as “H&E staining”) is normally used especially for a pathological specimen.
The hematoxylin is a natural substance extracted from a plant and does not have a nature of staining in itself. However, a hematin as an oxide of the hematoxylin is a basophilic stain and combines with a negatively charged substance. Since a deoxyribonucleic acid (DNA) included in a cell nucleus is negatively charged due to a phosphate group included as a constituent, the deoxyribonucleic acid combines with the hematin and is stained bluish violet. Since it is common to tend to use the hematoxylin as a stain name, the description below will be made by following this tendency though it is not the hematoxylin but the hematin as the oxide of the hematoxylin that has the nature of staining as described above. On the other hand, the eosin is an acidophilic stain and combines with a positively charged substance. It depends on a pH environment whether an amino acid and a protein are charged positively or negatively, and an amino acid and a protein have a tendency of being charged positively under an acid environment. Therefore, an eosin solution to which an acetic acid is added is sometimes used. A protein included in a cell nucleus combines with the eosin and is stained red or pink.
The specimen (stained specimen) after the H&E staining can be visually recognized with ease since a cell nucleus, a bone tissue, and the like are stained bluish violet and a cell cytoplasm, a connective tissue, a red blood cell, and the like are stained red. As a result of this, an observer can grasp a size and a positional relationship of constituents including the cell nucleus which constitute the organism, and can morphologically judge a condition of the stained specimen.
The observation of the stained specimen can be performed by displaying the stained specimen obtained through a multiband imaging on a display screen of an external device, except for the visual check by the observer. In the case of displaying on a display screen, a processing of estimating a spectral transmittance at each point of the specimen based on the captured multiband image; a processing of estimating an amount of the stain staining the specimen based on the estimated spectral transmittance; a processing of correcting a color of the image based on the estimated stain amount; and the like are performed. Then, characteristics of a camera, a dispersion of a stain condition, and the like are corrected, so that an RGB image of the specimen is composed for the display. FIG. 13 shows an example of the composed RGB image. When the stain amount is appropriately estimated, a specimen with a deep stain and a specimen with a light stain can be corrected as an image whose color is comparable to an appropriately stained specimen. Therefore, estimating a spectral transmittance of a stained specimen with high precision results in realizing a high precision in estimating an amount of a stain fixed to the stained specimen, correcting a dispersion of the staining, and the like.
As a method of estimating the spectral transmittance at each point of the specimen from the multiband image of the specimen, for example, an estimation method through a principal component analysis (see “Development of support systems for pathology using spectral transmittance—The quantification method of stain conditions”, Proceedings of SPIE, Vol. 4684, 2002, p. 1516-1523, for example), an estimation method through a Wiener estimation (see “Color Correction of Pathological Images Based on Dye Amount Quantification”, OPTICAL REVIEW, Vol. 12, No. 4, 2005, p. 293-300, for example), and the like can be quoted. The Wiener estimation, which is widely known as one of linear filtering methods of estimating an original signal from an observed signal on which a noise is superimposed, is a method of minimizing errors by taking statistical properties of an observation target and properties of a noise (observed noise) into consideration. Since a signal from a camera includes some sort of noise, the Wiener estimation is fairly useful as the method of estimating an original signal.
Here, a method of estimating a spectral transmittance at each point of a specimen based on a multiband image of the specimen through the Wiener estimation will be explained.
First, a multiband image of a specimen is captured. For example, by using a technology disclosed in Japanese Patent Application No. H7-120324, a multiband image is captured via a frame sequential method while rotating and switching sixteen band-pass filters by a filter wheel. By this, a multiband image having pixel values for sixteen bands at each point of the specimen can be obtained. Though a stain essentially ranges in three dimensions within a stained specimen as an observation target, the stain cannot be captured as a three dimensional image as it is in a normal transmission observation system and is observed as a two dimensional image obtained through a projection of illumination light passing through the specimen on an imaging element of the camera. Therefore, each point described here means a point, corresponding to each pixel of the projected imaging element, on the specimen.
With respect to a given point x in the captured multiband image, a relationship expressed by the following equation (1) based on a response system of the camera is true between a pixel value g(x, b) in a band b and a spectral transmittance t(x, λ) at a corresponding point on the specimen.g(x,b)=∫λf(b,λ)s(λ)e(λ)t(x,λ)dλ+n(b)  (1)A symbol λ represents a wavelength, a symbol f(b, λ) represents a spectral transmittance of a No. b filter, a symbol s(λ) represents a property of a spectral sensitivity of a camera, a symbol e(λ) represents a property of a spectral radiation of an illumination, and a symbol n(b) represents an observation noise in the band b. The symbol b, which is a serial number for identifying a band, is an integer value satisfying 1≦b≦16.
In an actual calculation, the following equation (2) obtained via a discretization of equation (1) in a wavelength direction.G(x)=FSET(x)+N  (2)
Provided that the number of samples in the wavelength direction is D and the number of bands is B (here B=16), a symbol G(x) represents a matrix which is formed by B rows and one column and deals with the pixel value g(x, b) at the point x. In the same way, a symbol T(x) represents a matrix which is formed by D rows and one column and deals with the t(x, λ), and a symbol F represents a matrix which is formed by B rows and D columns and deals with the f(b, λ). On the other hand, a symbol S represents a diagonal matrix which is formed by D rows and D columns and its diagonal elements deal with the s(λ). In the same way, a symbol E represents a diagonal matrix which formed by D rows and D columns and its diagonal elements deal with the e(λ). A symbol N represents a matrix which is formed by B rows and one column and deals with the n(b). In equation (2), since formulas with respect to multiple bands are aggregated by using matrices, a specific number for the “b”, which is variable and represents what band it is, is not explicitly described. Besides, an integral with respect to the wavelength λ is replaced by a product of the matrices.
Here, a matrix H defined by the following equation (3) is introduced to make the description easier. This H is also called “system matrix”.H=FSE  (3)
Next, a spectral transmittance at each point of the specimen is estimated from the captured multiband image by using the Weiner estimation. An estimation value {circumflex over (T)}(x) of the spectral transmittance can be calculated by the following equation (4). It should be noted that the symbol {circumflex over (T)} means that a hat (^) representing an estimation value is attached on top of T.{circumflex over (T)}(x)=WG(x)  (4)
Here, a symbol W is expressed by the following equation (5) and called “Wiener estimation matrix” or “estimation operator used for the Wiener estimation”. In the following explanation, the W is simply referred to as “estimation operator”.W=RSSHt(HRSSHt+RNN)−1  (5)where ( )t means a transposed matrix and ( )−1 means an inverse matrix.
A symbol RSS represents a matrix formed by D rows and D columns and represents an autocorrelation matrix of the spectral transmittance of the specimen. In addition, a symbol RNN represents a matrix formed by B rows and B columns and represents an autocorrelation matrix of a noise of a camera to be used for imaging.
When the spectral transmittance {circumflex over (T)}(x) is estimated in this way, a stain amount at a corresponding point of the specimen is estimated based on the {circumflex over (T)}(x). As an estimation target, there are three kinds of stains, which are a hematoxylin that, corresponding to a first stain amount, stains the cell nucleus; an eosin that, corresponding to a second stain amount, stains the cell cytoplasm; and an eosin that, corresponding to a third stain amount, stains the red blood cell. Here, the hematoxylin is abbreviated as a stain H, the eosin staining the cell cytoplasm as a stain E, and the eosin staining the red blood cell as a stain R, respectively. Strictly speaking, the red blood cell even in a state of not being stained has an inherent color in itself and is observed with the color inherent to the red blood cell and the color of the eosin altered in the staining process overlapped after the H&E staining. Therefore, a color produced in combination with both colors is called as the stain R, to be precise.
It is generally known in a light transmissive substance that a Lambert-Beer law expressed by the following equation (6) is true between an intensity I0(λ) of incident light and an intensity I(λ) of outgoing light at each wavelength λ.
                                          I            ⁡                          (              λ              )                                                          I              0                        ⁡                          (              λ              )                                      =                  ⅇ                                    -                              k                ⁡                                  (                  λ                  )                                                      ·            d                                              (        6        )            
A symbol k(λ) indicates a value which is specific to a substance and determined depending on the wavelength, and a symbol d means a thickness of the substance. Besides, the left side of equation (6) indicates the spectral transmittance.
When the H&E stained target specimen is stained by the three kinds of stains, i.e., the stain H, the stain E, and the stain R, the following equation (7) based on the Lambert-Beer law is true at each wavelength λ.
                                          I            ⁡                          (              λ              )                                                          I              0                        ⁡                          (              λ              )                                      =                  ⅇ                      -                          (                                                                                          k                      H                                        ⁡                                          (                      λ                      )                                                        ·                                      d                    H                                                  +                                                                            k                      E                                        ⁡                                          (                      λ                      )                                                        ·                                      d                    E                                                  +                                                                            k                      R                                        ⁡                                          (                      λ                      )                                                        ·                                      d                    R                                                              )                                                          (        7        )            
Here, each of symbols kH(λ), kE(λ), and kR(λ) represents a symbol k(λ) corresponding to each of the stain H, the stain E, and the stain R. Besides, each of symbols dH, dE, and dR is a virtual thickness of each of the stain H, the stain E, and the stain R at each point, corresponding to each image position of the multiband image, of the specimen. Though a concept of the “thickness” is not accurate since a stain is essentially present by dispersing over the specimen, the thickness works as a marker, indicating how much amount of stain is present, for a relative amount of stains, compared to a case of assuming that the specimen is stained with a single stain. In other words, it is possible to say that each of the dH, dE, and dR represents each amount of the stain H, the stain, E, and the stain R. By preparing a specimen stained with a single stain in advance and measuring the spectral transmittance thereof by using a spectrometer, the kH(λ), kE(λ), and kR(λ) can be easily obtained based on the Lambert-Beer law.
An extraction of a logarithm from both sides of equation (7) is expressed by the following equation (8).
                                          -            log                    ⁢                                    I              ⁢                              (                λ                )                                                                    I                0                            ⁡                              (                λ                )                                                    =                                                            k                H                            ⁡                              (                λ                )                                      ·                          d              H                                +                                                    k                E                            ⁡                              (                λ                )                                      ·                          d              E                                +                                                    k                R                            ⁡                              (                λ                )                                      ·                          d              R                                                          (        8        )            
When an element corresponding to the wavelength λ of the spectral transmittance data {circumflex over (T)}(x) estimated by using equation (4) is {circumflex over (t)}(x,λ) and the {circumflex over (t)}t(x,λ) is substituted into equation (8), the following equation (9) is obtained.−log {circumflex over (t)}(x,λ)=kH(λ)·dH+kE(λ)·dE+kR(λ)·dR  (9)
Here, since the dH, dE, and dR are unknown variables in equation (9), when simultaneous equations are set up from equation (9) with respect to at least three different wavelengths λ, they can be solved. To further enhance the accuracy, simultaneous equations may be set up from equation (9) with respect to four or more different wavelengths λ and a multiple regression analysis may be performed.
Then, when the stain amount at a point of the specimen is estimated in this way, the stain amount at each point of the specimen is adjusted to an appropriate stain condition based on the estimated stain amount and an image of the specimen is composed.